Computer Graphics

GUIs

Graphical User Interfaces (GUIs)

Development Tools

What are Development tools? :

· Tools that help a developer convert interface specifications into an interactive system and that support all phases of system refinement including prototypes, implementation, testing, maintenance, and enhancement

Conventional Development Tools

• Window Systems
• Constrain each application or application component to a particular frame on the screen (the window)
•  Provide an:
• Input model - user interaction sent to application as events
• Graphic model - output graphics
• Window management - window opened and closed, resized, moved, and iconified
• Presentation stype enforecement - e.g. X, Mac, Win95, Win3.1

e.g. X Windows:

• Network-based - uses client-server architecture -Any Xapp, called a client, runs on any machine on the net
• Server process, which resides on workstation, mediates all usr input and display output
• Server translates person's input into X events and sends to appropriate client
• Client interprets events and sends graphics direct to server on how to update display
• Special client, called Window Manager, is responsible for overall look and feel

BUT:

• Windowing systems provide too low-level an abstraction for building an application
• Better - have toolkits, interface builders, user interface mgmt. Systems.

Toolkits - a variety of interface building blocks called widgets and language for putting together

e.g. Motif, Xview, Tk/Tcl , Java AWT

· Widgets - buttons, menus, sliders, scrollbars, text editors, canvases for graphics, terminal windows

· Interfaces are built by using widgets, and programming languages (e.g Tk/Tcl)

· Widget Limitations:

o Widget sets are small and restrict interaction techniques

o Widgets are good for creating control panels BUT poor for what is seen inside. (e.g. Tk/Tcl for visualization resorts to OpenGL for graphics)

o Programming in an abstract language is not easy way for putting widgets together.

Motif Examples: (from: http://www.cm.cf.ac.uk/Dave/X_lecture/X_book_caller/index.html )

(Common Desktop Environment (CDE))

Front Panel

Drawing Widget

File Selection Box Widget

Interactive Builders - provide a direct manipulation of graphical and interface toolkits. Some are integrated into toolkit environment, e.g.

• NeXT interface builder
• Visual Basic
• XF builder for Tk/Tcl
• Visual Tcl : http://vtcl.sourceforge.net/
• Visual Gipsy for Tcl : http://www.prs.de/int/products/vg/index.html

• Implementer selects a widget from a palette, places it on the screen, sets its attributes through form filling, or direct manipulation. Links made as callbacks that are written as procedures.
• Allows developers to test interfaces as they are being constructed.
• Also can build interface prototypes with Macromedia Director and HyperCard

Examples:

• XF Interface:

• Visual Gipsy Interface

• Visual Tcl

User Interface Management Systems (UIMS) - better said as "user interface development environments."
Separate Applications into 3 modules:

• Presentation Graphics - output to screen.
• Dialog Control - how input is passed to Applications
• Application Interface - how interface hooks into application code

Good - Application and Interface separated
Bad - Hard for designer to separate two in practice

e.g. SUIT (Simple User Interface Toolkit):
Can learn to build an interface in a few hours.
Designed for teaching.
Adv:

• Portability across platforms
• Standard widget set
• Runs applications plus application builder
• Allows for custom widgets

Novel Approaches to Developer tools

Garnet - System and Tools - comprehensive toolkit:

• Easy manipulation of graphical objects and behavior
• Can create and edit application specific graphical objects and predefined widgets
• All objects are defined in Opal graphical toolkit
• Opal implemented atop KR - prototype instance object oriented system
• If prototype is edited - all objects changed
• Constraints
• Constrain-based tooled that allows developer to specify relationships among components (therefore at runtime system makes sure changes in one object are made in others)
• Interactors
• Separate look from feel allows many different styles of menus.
• Programming by demonstration (PBD)
• Examples of the interface are created and then the program is generated.

Sample Applications of Garnet:
http://www.cs.cmu.edu/afs/cs/project/garnet/www/screen-shots/screen-shots.html

Amulet - Garnet's Successor
http://www.cs.cmu.edu/afs/cs/project/amulet/www/amulet-home.html

User interface development environment for C++ and is portable across X11 on all kinds of Unix (Sun, Dec, HP, SGI, Linux, NetBSD, etc.), Microsoft Windows 95 and NT, and the Macintosh.

• Amulet helps create graphical, interactive user interfaces.
• Amulet includes features designed to make creation of highly-interactive, graphical, direct manipulation user interfaces significantly easier employing:

• prototype-instance object model
• constraints
• high-level input handling including automatic undo
• built-in support for animation
• gesture-recognition
• full set of widgets.

Tools for Entire Development Cycle

• Toolkits should support
• Interface specification
• Implementation
• Testing

Alternate Approach - instead of "artifact centered" of existing tools USE "semantically driven user interface design."

Semantic model - interface developed as a declarative description and model becomes a plan that is used to let the developer manipulate in a way that is appropriate to its stage in the lifecycle.

e.g. IBM's Interactive Transaction System (ITS)

• Action layer - implements semantics of application functions independent of interface.
• Dialog layer - interface is a set of logica; frames containing objects that define flow of control between frames and actions.
• Style rules - defines the presentation and behavior of interactive widgets.
• Style programs - implement actual widgets.

NOTE: Each layer corresponds to work roles of the development team members:

• Application programmer - works with action layer
• Application expert - composes dialog rules
• Style programmers - build style programs
• Graphic designers - compose style rules

Groupware Toolkits
Supports and augments group work where conventional toolkits do not work
e.g. Groupkit Project - University of Calgary
http://www.cpsc.ucalgary.ca/grouplab/projects/GroupKit.html

• Used to build real-time applications such as drawing tools, editors and meeting tools that are shared simultaneously among several users.
• Used for prototyping groupware, investigating multi-user architectures and interfaces, and as a CSCW teaching tool.

Example:

• Diagram at right and code below fully define a "Hello World" program written in GroupKit.
• When one person presses the hello button, all people in the conference see that person say hello!

1.gk_initConf $argv
2.gk_defaultMenu .menubar
3.pack .menubar -side top -fill x
4.set greetings "[users local.username] says hello!"
5.button .hello -text "Hello World" \

-command "gk_toAll say_hi
\"$greetings\""

6.pack .hello -side top
7.proc say_hi {new_label} {
8. .hello configure -text $new_label
9. after 2000 {.hello configure -text "Hello World"}
10.}

Example:

SharedNotes is a system that allows people to create and manipulate both personal and public notes between three devices: a personal digital, assistant (PDAs, in this case the Palm Pilot), a large public display, and their office computer (which acts both as a personal device and a system for remote collaboration).

• The left figure shows how a person would see their personal and public information on their PDA, while the right figure shows how this same information is reflected in the public display.
• Essentially, people can work on their PDAs or computers for personal work, bring some or all of their work into a meeting using a public display, work on the public display either directly or indirectly through their PDAs, and walk away with a record of all public contributions.

Touch, Gesture, and Marking

Haptic input (Greek for contact) - involves physical contact between computer and user.

• Hands on mouse
• Foot on pedal
• Tongue with joystick

Relating Task and Technology

• Designers must find match between application and input technology.
• Most recognize the relevant dimensions along which an application's demands can be characterized
• Must know how each technology performs along those dimensions

Basic set of generic tasks:

• Pursuit tracking

Test:  Fly moves over screen under computer control

• Operator uses control device to track fly's motion
• Test how many times fly can be swatted in a given time interval
• Parametrics: speed target moves
• Control: display ratio (C:D)
• Ratio between distance the controller must be moved to cause tracker to move given distance on display.

e.g. C:D 2:1, 2 cm of controller results in 1 cm. on screen.

• C:D ratio can be a variable that changes as controller moves
• Button push is another parameter - good on mouse more, difficult on tablet

• Target Acquisition

Test: User selects each of a number of rectangles displayed on screen

• selected by positioning tracking symbol and signaling with button
• statistics: how long it takes to select full set of targets
• check out target size effect on speed
• Fitts law: MT = a + b log2 (D/W + l)

Movement time (MT): time to move the hand to a target of width (W) which lies (D) distance away, where a is a constant and b=100 [70 - 120] msec/bit.

• Variations on task:
• Homingtime - moving between text entry device and pointing device

- time it takes (test by having user hit space bar after clicking on rectangle)

• Number of dimensions - higher dimensional tasks

What is effective target within two dimensions or more?
What is effect of approach angle?

• Dragging vs. rubber-band lines - object is dragged from square to square.

 - Also Fitts law task -> Holding mouse down can effect target acquisition

• Left hand vs. right hand - moving from dominant to non-dominant varies across devices.
• Free-hand inking - attempt to write signature with different technologies.
• Tracing and digitizing -  CAD, cartography, graphic arts

- Relative devices useless
- Absolute devices vary widely
- Resolution important (eg cartography)

• Constrained motion -  must be able to move tracker rapidly over a straight-line path.
  e.g.  - scrollbar mechanism

- move mouse in x or y

- Thumbwheels work better than a mouse

- Task: tracking symbol is a ball that is dragged along a linear path without crossing parallel lines.

• How is speed affected by input device?
• How is speed affected by path width?
• How about horizontal vs. vertical?
• What about circular motion ?

• 3D Input and Interaction:
• Need 3D equivalents of 2D tasks.
• What devices support applications of higher dimensionality?
• Can support 3D with lower devices (eg. virtual 3D trackball).

A Taxonomy of Input Devices

• Categorization

• Table uses hierarchy of criteria.
• Generality and Extensibility: tradeoff between the generality of a computer workstation and groups of specialized tasks it is used for. THUS: No optional device.
• Relative vs. absolute controls: affects dialogs.
• What our taxonomy does not show: Only considers continuous devices.

Chunking and Phrasing

• Human -Computer dialogs can benefit from appropriate phrasing such as found in music or speech.
• Most human computer dialogs are composed like speech: selecting/positioning, positioning/scaling, navigating/scaling.
• Structure that emerges from appropriate phrasing can accelerate the process.

e.g proof-reading marks

• Novices "chunk" together concepts.
• Modes and Mode Errors:
• Mode error: misclassification of a situation resulting in actions that are inappropriate for the true situation.

Marking
Style of interaction (gesture driven, e.g. pen-based)

• User's input is stream x,y coordinates called "digital ink", thus marking interfaces
• Most marking systems are different to use and prone to mode errors.

Who Does the Recognition?

· Recognition is heart of marking system: block characters, cursive script, etc.

· Pattern recognition is hard and still unsolved.

· Should focus on systems where marks are recognized by user.

What is Recognition?

· High level marks, such as proofreaders, easier to learn.

Self-Revelation and Marking Menus:

• Marking systems assume a form-filling style. May not have enough space, such as on PDA.
• Alternatively marking-menus:

• User presses down, waits until a pie menu appears and strikes through slice.
• Alternatively, mark-ahead by making a mark without waiting for menu. System recognizes physical movement.

Working within the Marking Idiom:

• Problems when designers do not adequately conceptualize the medium.
• Free-form text sometimes faster than keyboard, 2² & 22 easier drawn.
• But if text is entered on a line, using text with a graphical keyboard and stylus is faster and more accurate (eg. 0 vs. O, number vs. letter).
• PDA too small to vie with graphical keyboard but could use some shortcuts.
• e.g .Uni-strokes alphabet

• e.g. With mnemonics

Strengths:

· Can take about one hour to learn.

· Only single stroke.

· No segmentation problem - which stroke belongs to what.

· Can recognize character when each one is written upon other.

· Marking system leave an explicit audit trail of user's actions.

· Figure/ground relationship on computer - know which marks made by computer and which by person

Situated Design

• If electronic version of system is not faster than pen and paper then redesign system.

Gestures

• Some systems "gesture" refers to marking interfaces.
• Other system "gesture" itself is recognized.

e.g. Myron Krueger - Videoplace - non-invasive (video camera, image processing)

• The "Videoplace" system perceives one or more participants and responds to their movements in realtime using video cameras.
• The "Videoplace" system identifies each participant's head, arms, legs, hands and fingers and determines their rate of movement.
• Each participant's image may be moved, scaled or rotated anywhere on the screen.
• "Videoplace" has two modes:
• It interacts directly with an individual.
• It mediates a dialog  between two participants: one who understands and controls the system and a second naive participant.  The controller can interact with the other person by using the image of his or her hands.

Gestures in Collaborative Work

• Remote awareness problem arises in collaborative meetings, etc.
• How do you include hand motion?
• Capture image of hands over work surface.
• Then add Kruger's recognition.

Two-handed Input

• Real world use of two hands: painting, threading a needle, taking notes, driving a car, preparing food.
• Asymmetrical bimanual action:

Three properties of bimanual asymmetric actions:

• the non-dominant hand determines the frame of action of the dominant hand.

e.g. holding nail to be hammered.
       holding needle to be thread.

• the sequence of action is non-dominant hand then dominant.

e.g. above examples.

• the action of the non-dominant hand is coarse relative to the fine action of the dominant.

• People spontaneously use two hands in solving problems.
• Demonstration: - position and scaling

 - navigation and selection

New paradigm: see-through interface

• Uses toolglass and magic lens:

http://www.parc.xerox.com/istl/projects/MagicLenses/93Siggraph.html

• 2 1/2 D functions on 3 planes:
• desktop where icon sits.
• cursor floats above desktop and icons - manipulated with dominant hand with mouse

• magic lens and tool glass sheets - lie between cursor and desktop.
• kind of like protractor and rulers.
• you can see tool and its markings and see paper below.
• They get moved with non-dominant hand using trackball or small touch table.

• Relationship between levels:

§ Toolglass sheet with click through buttons.

§ Magic lenses are visualization widgets analogous to magnifying glasses with diverse optical properties.

e.g. An achromatic lens over a drop shadow lens over a knotwork.

e.g. The local scaling lens

Realizing Inputs Full Potential

• Need experience

Transparent Access and the Physically Disabled

• Mouse becomes tongue activated joystick.
• Button replaced by blow/suck tube.
• Need more transparency.

Device Independence and Virtual Devices

• Generic virtual devices

Application written in device independent way all it needs to know is kind of input (e.g. text).

• Logical Input Devices
• Locator – a device for specifying a coordinate position (x,y)
• Standard method for interactive selection by positioning cursor
• Mouse, joystick,thumb-wheels, dials, stylus, etc.
• Keyboard cursor control keys
• Stroke – a device for specifying a series of coordinate positions.
• Multiple calls to locator device – used to display line segments
• Valuator – a device for specifying scalar values
• Set of control dials, sliders
• Pick – a device for selecting picture elements
• Select parts of a scene – pick window
• String – a device for specifying input text
• Keyboard input
• Choice - a device for selecting menu options

Input Functions

Specify the following:

· Which physical devices are allowed

· How program and devices interact

· How data is input

Input Modes – how program and input devices interact

· Request Mode – application program initiates data entry

o Data requested => process suspended until data entered

o Device put into wait state until input request made

· Sample Mode – program and input devices operate independently

o Program may run simultaneously with input devices

o When program needs new data it samples current data

· Event Mode – input devices initiate data input to application program